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SECTION  I 
INTRODUCTION 


Northrop  Corporation,  Electro-Mechanical  Division,  is  pleased  to  submit  the  second 
quarterly  report  on  the  Experimental  Development  of  a FLIR  Sensor  Processor  program. 
During  the  first  quarter,  the  test  system  configuration  was  assembled  and  checked 
out,  the  software  debugged,  and  Task  I of  the  contract,  the  Experiment  and  Study 
Plan,  completed.  The  first  quarter  report  was  submitted  and  a technical  review  was 
held  at  Northrop  October  18,  1976. 

\ 

This  report,  which  covers  the  work  performed  during  the  second  quarter,  primarily 
concerns  the  autofocus  experiments.  The  object  was  to  determine  the  use  of  the  2D 
Haar  processor  as  an  accurate  sensor  of  in-focus/defocus  conditions  in  FLIR  systems. 
To  validate  this  use,  a series  of  measurements  were  taken  to  determine  sensitivity, 
accuracy,  and  repeatability  (temporarily  substituting  a camera  for  the  FLIR).  Ini- 
tial measurements  were  made  using  a RETMA  resolution  chart.  This  was  followed  by 
using  thermograms  provided  by  NVL.  The  analysis  and  graphed  results  show  the  effect 
of  focus  adjustment  on  the  ac  magnitudes  and  ac  power  of  the  Haar  coefficients  and 
the  generalized  descriptors.  Using  imagery  with  some  amount  of  activity  within  the 
view,  it  can  be  shown  that  a simple  scalar  value  can  be  computed  in  real  time,  which 
has  a sensitivity  greater  than  the  eye  in  determining  best  focus. 


The  second  part  of  this  report  discusses  the  automatic  deblurring  experiments,  which 
have  already  been  started  and  are  planned  to  continue  into  the  following  quarter.  *r 
They  Involve  analog  and  computational  simulations,  an  experimental  evaluation  tech-  \ 
nlque,  alternative  approaches  for  estimating  unknown  blur  functions,  and  the  appli-  \ 
cation  of  these  functions  to  deblurring.  Application  of  the  2D  Haar  processor  to 
blur  estimation  and  deblurring  is  fully  discussed.  Other  experiments  in  image  en- 
hancement, adaptive  signal  processing,  and  data  compression  have  been  started  but 
have  not  been  described  in  this  report. 
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SECTION  2 

AUTO  FOCUS  EXPERIMENTS 


2 . I INTRODUCTION 

Because  severe  demands  are  placed  on  a two-man  helicopter  team,  the  need  for  manual 
FLIR  refocusing  can  pose  a formidable  problem.  FLIR  defocusing  arises  from  continual 
changes  in  observation  range,  small  thermal  deformations  of  the  lens,  and  other 
incidental  causes.  Automatic  refocusing  offers  a means  to  combat  these  tendencies 
and  relieve  part  of  the  manpower  burden.  The  apparatus  must  sense  the  defocus  con- 
dition of  the  image  and  produce  a corresponding  error  signal  for  automatic  lens  ad- 
justment. The  objective  of  this  experiment  is  to  develop  a measure  of  the  state  of 
focus  of  an  image  based  on  the  distribution  of  energy  in  its  rationalized  Haar 
transform.  The  resultant  focus  measure  may  then  be  used  as  the  basis  for  the  gen- 
eration of  a focus-correcting  error  signal. 

The  focus  measure  developed  should  exhibit  the  following  characteristics  in  order 
to  be  compatible  with  system  and  operational  requirements: 

a.  Low  Computational  Complexity  - The  amount  of  computation  required  to 
evaluate  an  image's  state  of  focus  must  be  amenable  to  real-time  (TV  rates) 
processing. 

b.  High  Sensitivity  to  Focus  - Small  changes  in  an  image's  state  of  focus 
should  produce  significant  changes  in  the  focus  measure. 

c.  Noise  Immunity  - In  the  present  test  configuration,  noise  in  the  video 
signal  before  digitization  is  the  overriding  contributor  to  system  noise.  In  an 
operational  configuration,  small  (one  pel)  translations  of  the  image  may  also  pro- 
duce a significant  amount  of  noise.  The  focus  measure  should  be  relatively  insen- 
sitive to  both  types  of  image  noise. 
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d.  Scene  Independence  - The  focus  measure  should  perform  equally  well  on 
both  background  and  targets;  however,  there  are  some  limitations  to  this  requirement. 
There  must  be  sufficient  activity  in  the  area  of  Interest  to  evaluate  the  image's 
state  of  focus.  For  example,  any  focus  measure  will  not  be  expected  to  perform  well 
when  the  image  contains  only  clear  sky. 

e.  Adaptability  in  an  Operational  System  - The  focus  measure  in  its  final 
form  must  be  able  to  serve  as  the  basis  for  the  generation  of  an  error  signal  in  a 
servo  system.  This  signal  must  have  sufficient  bandwidth  and  stability  to  be  com- 
patible with  FLIR  image  quality  parameters  and  ASH/AAH  operational  requirements. 

These  specific  system  requirements  are  discussed  in  the  Appendix  to  the  first 
quarterly  report  (76Y133). 

The  experiment  is  divided  into  two  major  sections.  In  the  first  section  the  per- 
formance of  several  candidate  focus  measures  is  examined.  The  test  image  used  in 
this  section  is  a video  test  pattern  (RETMA  resolution  chart).  Tradeoffs  of  the 
characteristics  of  the  candidate  focus  measures  are  made  and  two  of  them  are  selected 
for  subsequent  analysis.  In  the  second  section,  the  capabilities  of  the  selected 
focus  measures  are  evaluated  on  the  set  of  five  FLIR  thermograms  supplied  by  NVL. 

Both  target  and  background  objects  are  used  as  input  data  in  order  to  determine  the 
behavior  of  the  focus  measures  on  general  classes  of  images. 

2.2  GENERAL  EXPERIMENTAL  PROCEDURE 

The  experimental  test  configuration  is  shown  in  Figure  2-1.  Test  imagery  consisted 
of  a video  test  pattern  and  five  FLIR  thermograms  supplied  by  Night  Vision  Labora- 
tory (NVL)  (Figure  2-2).  The  focus  of  the  video  camera  was  set  manually  with  the  aid 
of  13  equally  spaced  marks  on  the  lens  barrel.  Figure  2-3  shows  the  optical  rail 
apparatus  and  a more  detailed  view  of  the  video  camera  lens. 

The  data  collection  sequence  for  each  of  the  focus  measures  was  as  follows:  The 

video  camera  lens  was  set  initially  at  one  extreme  (lens  position  A) . The  Haar 
transforms  of  both  fields  of  four  previously  selected  sectors  of  the  image  were 
acquired  and  the  candidate  focus  measure  computed.  The  result  was  then  printed  out 
on  the  Centronics  Line  Printer. 
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Figure  2-1.  EXPERIMENTAL  TEST  CONFIGURATION 


(E)  MAGE  4 (F|  (MAGES 

Figure  2-2.  TEST  IMAGERY  (Sheet  2 of  2) 

This  was  repeated,  advancing  the  lens  position  one  step  each  time,  until  the  other 
extreme  lens  position  (lens  position  M)  was  reached.  The  entire  process  was  then 
repeated,  stepping  the  lens  backward  from  position  M to  position  A.  This  process 
of  advancing  and  retracting  the  lens  position  through  one  cycle  while  evaluating  a 
focus  measure  at  each  position  will  be  referred  to  as  a "Lens  Cycle."  One  lens 
cycle  takes  about  6 minutes  to  complete. 


Noise  measurements  of  each  focus  measure  were  performed  by  constantly  computing  the 
focus  measure's  value  while  leaving  the  video  camera  lens  stationary. 

2.3  TEST  PATTERN  EXPERIMENT 

2.3.1  Candidate  Focus  Measures 

Several  preliminary  measures  of  an  image's  state  of  focus  were  selected  for  study. 
Each  measure  is  based  on  the  distribution  of  energy  in  the  Haar  transform  of  the 
image.  The  candidate  focus  measures  are  presented  here  in  order  of  increasing  com- 
putational complexity. 

2. 3. 1.1  Sum  of  the  Magnitudes  - The  sum  of  the  magnitudes.  S . of  the  ac  Haar 

m 

coefficients  is  expressed  as 

s - y j y y |a  | -a  a> 

m l l l l i rmpn 1 -1,1, -1,1  K ' 

r m p n 
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Figure  2-3.  AUTOFOCUS  TEST  SETUP 


S is  the  computationally  simplest  index  of  the  total  ac  information  in  an  image's 
m 

Haar  transform.  In  order  to  ascertain  the  distribution  of  ac  information  through- 
out the  transform  as  well,  totals  of  the  magnitudes  of  the  coefficients  were  also 
taken  for  each  sequency  group. 


2. 3. 1.2  Sum  of  the  Normalized  Magnitudes  - The  sum  of  the  normalized  magnitudes, 

S , of  the  ac  coefficients  is  expressed  as 
n 


S - l l l l f |a  | 
n l l l l rp  rmpn 

r m p n 


-a 


(2) 


-1,1, -1,1 

The  two  dimensional  sensor  processor's  implementation  of  the  rationalized  Haar 
transform  contains  an  inherent  high  sequency  boost.  To  remove  this  boost,  a nor- 
malization factor,  f , is  introduced  to  weight  the  information  from  the  rpC^ 
sequency  group.  The  sum  of  the  normalized  magnitudes  more  closely  approximates 
the  sum  of  the  magnitudes  of  an  unrationalized  Haar  transform.  S was  evaluated 
for  all  the  ac  terms  as  well  as  for  each  sequency  group. 


2. 3. 1.3  AC  Power  - The  ac  power,  C,  is  defined  as 

c - mi  (■„,/  - <*_!,!.-!. !>2  O) 

rmpn 

The  parameter  C is  equivalent  to  the  local  activity  of  a 16  by  16  pel  image  sector. 
The  image's  activity  should  provide  an  index  of  focus  that  is  more  noise  immune  and 
more  sensitive  to  focus  than  the  previous  measures.  The  noise  immunity  arises  from 
the  fact  that,  by  squaring  terms,  a few  large  fluctuations  in  amplitude  may  still 
be  seen  despite  the  presence  of  many  small  ones.  The  increased  sensitivity  to 
focus  is  due  to  a similarity  between  the  Fourier  and  unrationalized  Haar  transforms. 
Since  both  transforms  are  unitary  and  possess  equivalent  dc  terms,  Parseval’s  theorem 
tells  us  that  the  ac  energy  in  each  is  also  equivalent.  Defocusing  an  image  effec- 
tively subjects  the  image  to  a low-pass  filter,  thereby  decreasing  the  ac  energy  in 
its  Fourier  transform.  The  ac  energy  in  the  image's  unrationalized  Haar  transform 
would  likewise  be  decreased.  Since  the  rationalization  of  the  Haar  transform  in- 
volves only  simple  scaling  of  each  sequency  group,  the  direction,  if  not  the  amount, 
of  change  in  the  ac  energy  of  the  rationalized  Haar  will  be  equivalent  to  that  of 
the  unrationalized  Haar  and  Fourier  transforms. 


2. 3. 1.4  Texture  and  Edge  Activity  - Texture  and  edge  activities,  as  defined  in 
the  first  quarter  report,  are  the  most  expensive  computationally.  They  are  included 
in  the  study  to  determine  if  variations  in  image  activity  due  to  changes  in  state 
of  focus  will  appear  primarily  as  variations  in  texture  or  edge  activity. 


2.3.2  Focus  Measure  Data 


Each  of  the  foregoing  focus  measures  was  computed  through  one  lens  cycle.  A repre- 
sentative portion  of  a lens  cycle  printout  is  shown  in  Figure  2-4 . Since  during 
each  lens  cycle  each  lens  position  is  encountered  twice,  the  two  resultant  focus 
measure  values  are  averaged  to  facilitate  graphical  presentation  of  the  data. 

The  range  of  defocus  encountered  through  one  lens  cycle  is  shown  in  the  series  of 
photographs  in  Figure  2-5,  A through  M.  The  behavior  of  the  first  three  candidate 
focus  measures  is  shown  as  a function  of  lens  position  in  Figures  2-6,  2-7,  and  2-8. 
The  noise  characteristics  of  each  measure  are  shown  in  Figures  2-9,  2-10,  and  2-11. 

As  can  be  seen  in  Figures  2-6,  2-7,  and  2-8,  each  focus  measure  tested  exhibited  a 
peak  value  at  lens  position  G.  In  order  to  determine  if  lens  position  G is  indeed 
the  sharpest  focus  position,  the  fine  detail  of  the  test  pattern  was  observed  under  a 
hand  lens.  Close  examination  of  the  wedges  in  the  test  pattern  (Figure  2-5)  shows 
that  lens  positions  F,  G,  and  H are  the  ones  that  yield  images  with  sharper  focus. 
Visually  discriminating  the  difference  in  focus  between  F,  G,  and  H is  next  to  im- 
possible, however.  Since  lens  position  G is  in  the  center  of  F,  G,  and  H,  and  the 
peaks  in  the  focus  measure  curves  are  distinct,  lens  position  G is  deemed  to  be  the 
"in  focus"  lens  position.  The  ability  of  the  focus  measure  to  evaluate  the  state  of 
focus  of  an  image  appears  to  rival  that  of  the  observer. 

The  division  of  energy  into  sequency  groupings  for  each  focus  measure  can  be  seen 
in  part  A of  Figures  2-6,  2-7,  and  2-8.  The  changes  in  the  ac  energy  of  the  Haar 
transform  due  to  movement  of  the  lens  are  localized  in  sequency  groups  11  through 
15,  and  groups  7 through  10.  Also,  both  sequency  groupings  tend  to  behave  in  a 
like  fashion.  It  is  possible,  then,  to  evaluate  the  state  of  focus  of  an  image 
using  only  the  information  contained  in  one  sequency  grouping.  Such  an  approach, 
however,  would  increase  the  focus  measure's  sensitivity  to  image  noise  (particularly 
image  motion)  as  well  as  allow  some  possibly  significant  changes  in  ac  energy  in 
another  group  to  go  unobserved.  For  these  reasons,  the  focus  measures  studied  here 
will  sum  all  of  the  ac  energy  within  the  image's  Haar  transform  and  will  not  single 
out  any  one  sequency  group  or  grouping. 

The  sharpness,  or  sensitivity  to  focus,  of  the  candidate  focus  measures  varies  sub- 
stantially. To  aid  in  comparison,  each  focus  measure  is  plotted  Jn  Figure  2-12. 

The  curve  resulting  from  the  sum  of  the  magnitudes  of  ac  coefficients  is  comparative- 
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Figure  2-4.  PORTION  OF  LENS  CYCLE  PRINTOUT 
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Figure  2-9.  RMS  NOISE  LEVELS  FOR  S , SUM  OF  AC  MAGNITUDES 
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Figure  2-10 
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Figure  2-11.  RMS  NOISE  LEVELS  FOR  C,  SUM  OF  AC  POWER 
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Figure  2-12.  A COMPARISON  OF  FOCUS  MEASURES 

ly  flat,  Its  performance  being  only  slightly  poorer  at  its  end  points  than  that  of 
the  sum  of  the  normalized  magnitudes  of  the  ac  coefficients.  The  relative  insen- 
sitivity exhibited  by  these  two  curves  is  due  to  the  uniform  weighting  of  each 
coefficient  within  any  sequency  group.  When  any  set  of  numbers  is  summed  in  this 
fashion,  a substantial  change  in  a few  members  of  the  set  often  is  obscured  by 
effectively  averaging  them  with  the  others.  By  squaring  the  numbers  before  summing 
them,  however,  the  sensitivity  to  small  changes  may  be  increased.  This  increased 
sensitivity  is  exhibited  in  Figure  2-12  by  the  curve  for  the  ac  power  contained  in 
the  image's  Haar  transform. 

Examination  of  the  noise  characteristics  of  each  focus  measure  (Figures  2-9,  2-10, 
and  2-11)  shows  that  their  noise  levels,  expressed  as  a fraction  of  peak  value,  are 
roughly  equivalent.  By  assuming  the  maximum  observed  value  of  a focus  measure  minus 
its  minimum  value  to  be  the  available  signal,  the  signal-to-noise  ratio  can  be 
determined.  The  signal-to-noise  ratios  of  the  first  three  focus  measures  are  shown 
in  Table  2-1.  As  expected,  C,  the  ac  power  of  the  image’s  Haar  transform,  has  the 
highest  signal-to-noise  ratio. 


2-20 


Table  2-1.  FOCUS  MEASURE  SIGNAL-TO-NOISE  RATIOS 


Signal -to-Noise 

Focus  Measure 

Ratio  (dB) 

S , Sum  of  AC  Magnitudes 
m 

43 

S , Sum  of  Normalized  AC  Magnitudes 

44 

C,  AC  Power 

49 

The  behavior  of  texture  and  edge  activity  as  a function  of  lens  position  was  deter- 
mined in  a fashion  similar  to  the  other  focus  measures,  i.e.,  evaluating  the  focus 
measure  at  each  point  through  one  lens  cycle.  However,  the  area  of  the  test  pattern 
for  which  the  descriptors  were  computed  was  different  than  that  used  for  the  other 
measures.  Figure  2-13  shows  the  test  pattern  with  the  descriptor  input  areas  flagged 
by  increased  brightness.  These  areas  were  chosen  so  that  texture  and  edge  would  be 
computed  for  both  the  center  of  the  image,  which  is  primarily  texture,  and  the 
borders  of  the  image,  which  are  composed  primarily  of  two  singular  edges. 

The  computed  values  of  texture  activity,  edge  activity,  and  their  sum  are  shown  as 
a function  of  lens  position  in  Figure  2-14.  As  depicted  in  Figure  2-14,  edge  activ- 
ity behaves  much  like  ac  power.  Texture  activity,  however,  exhibits  markedly  dif- 
ferent characteristics.  While  similar  to  the  other  curves  in  having  a peak  at  lens 
position  G,  the  texture  activity  curve  also  has  a local  minimum  at  lens  position  C. 
The  precise  effect  that  defocusing  an  image  has  on  the  division  of  activity  into 
texture  and  edge  information  is  not  yet  fully  understood. 

The  computational  complexity  of  each  of  the  focus  measures  is  tabulated  in  Table 
2-2.  The  computation  time  of  each  measure  is  the  time  required  for  the  instrumenta- 
tion system's  minicomputer  to  evaluate  the  state  of  focus  of  one  16  by  16  pel  block 
of  the  image.  The  minicomputer  (Computer  Automation  LSI-2)  has  an  average  instruc- 
tion cycle  time  of  1 usee.  System  overhead  operations  are  not  included  in  the  com- 
putation time  values. 

As  shown  in  Table  2-2,  S^  is  the  simplest  measure  to  compute  as  it  evolves  only  254 
additions  operations.  Sn  requires  190  power  of  two  multiplications  (shifts)  as 
well  as  the  additions  operations.  In  order  to  compute  C,  the  total  ac  pjwer  of  the 
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Figure  2-13.  TEST  PATTERN  DESCRIPTOR  INPUT  AREAS 


Figure  2-14.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION 
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Table  2-2.  FOCUS  MEASURE  COMPUTATIONAL  COMPLEXITY 


Measure 

Computation 

Time 

Real  Time 
Compatibility 

S^,  Sum  of  AC  Magnitudes 

S^,  Sum  of  Normalized  AC  Magnitudes 

C,  Total  AC  Power 

Texture  and  Edge  Activity 

254  psec 
464  usee 
765  usee 
~204  msec 

Excellent 

Good 

Good 

Very  Poor 

16  by  16  pel  block,  each  of  the  255  ac  coefficients  must  be  squared  before  summation 
occurs.  Because  of  the  LSI-2 's  arithmetic  format,  squaring,  or  simple  integer 
multiplication,  uses  two  computer  instruction  cycles.  The  number  of  cycles  re- 
quired to  compute  C then  is  (255  x 2)  + 254  » 765. 

Unlike  the  previous  three  focus  measures,  which  involve  integer  arithmetic,  texture 
and  edge  activity  require  the  use  of  floating  point  arithmetic.  As  a consequence, 
not  only  is  the  number  of  requisite  instruction  cycles  much  greater  for  each  opera- 
tion, it  is  data  dependent  as  well.  The  computation  time  for  texture  and  edge 
activity  listed  in  Table  2-2  is  the  time  observed  for  the  computation  of  these  meas- 
ures, scaled  down  by  an  estimate  of  percent  system  overhead.  The  texture  and  edge 
activity  focus  measure  is  clearly  not  compatible  with  a real-time  system. 

2.3.3  Summary 

The  performance  of  each  focus  measure  in  evaluating  the  state  of  focus  of  the  video 
test  pattern  is  summarized  in  Table  2-3. 

Of  the  measures  studied,  the  total  ac  power  provides  the  focus  measure  with  the 
best  sensitivity  to  focus  and  the  highest  signal-to-noise  ratio.  The  only  dis- 
advantage associated  with  using  ac  power  as  a measure  of  focus  is  that,  compared 

to  S and  S , it  is  relatively  expensive  to  compute.  In  terms  of  TV  frame  rates, 
m n 

however,  its  performance  is  still  acceptable.  For  example,  the  ac  power  of  about 
forty  16  by  16  pel  image  blocks  could  be  computed  during  the  elapsed  time  of  one 
TV  frame. 
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Table  2-3.  PERFORMANCE  OF  CANDIDATE  FOCUS  MEASURES 


Measure 

Sensitivity 
to  Focus 

Signal-to-Noise 
Ratio  (dB) 

Required 

Computation 

Time 

S , Sum  of  AC  Magnitudes 

Fair 

43 

254  usee 

S , Sum  of  Normalized  AC 
Magnitudes 

Fair 

44 

464  usee 

C,  AC  Power 

Good 

49 

756  usee 

Texture  vs  Edge 

Poor 

— 

204  msec 

The  measures  of  focus  to  be  evaluated  on  the  NVL  FLIR  Imagery  in  the  next  experi- 
ment are  C,  the  total  ac  power  or  image  activity,  and  X and  E,  the  texture  and  edge 
activities  of  the  image.  C was  selected  for  its  superior  performance  on  the  video 
test  pattern.  Texture  and  edge  activity  are  being  evaluated  further  to  determine 
their  behavior  on  images  that  are  of  a more  general  nature  than  the  video  test 
pattern. 

2.4  FLIR  IMAGE  EXPERIMENT 

In  order  to  ascertain  the  ability  of  ac  power  to  discriminate  focus  on  FLIR  Imagery, 
the  five  FLIR  images  supplied  by  NVL  were  used  as  input  data  in  the  test  configura- 
tion shown  in  Figure  2-1.  In  each  image,  areas  containing  only  background  and 
areas  containing  both  background  and  target  information  were  considered.  For  the 
background  data,  the  ac  power  of  three  fixed  sets  of  eight  16  by  16  pel  blocks  was 
evaluated.  These  input  areas  are  flagged  by  increased  brightness  in  Figure  2-15. 
Target  data  were  collected  by  restricting  the  input  data  to  only  the  sector(s)  that 
contained  one  of  the  targets  in  the  image  under  consideration. 

In  each  case,  the  ac  power  of  the  image  input  area  was  evaluated  through  four  lens 
cycles  and  the  results  averaged  for  graphical  presentation. 
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Figure  2-15.  FLIR  IMAGES  WITH  INPUT  AREAS  FLAGGED  BY  INCREASED  BRIGHTNESS 

(Sheet  2 of  2) 


2.4.1  Image  Data 


The  variations  in  ac  power  as  a function  of  lens  position  are  shown  in  Figures  2-16 
through  2-21  for  background  data,  and  in  Figures  2-22  through  2-26  for  target  data. 

As  can  be  seen  in  Figures  2-16  through  2-21,  the  background  data  curves  are  rela- 
tively insensitive  to  focus.  The  curve  for  Image  5 (Figure  2-20),  as  a worst  case 
example,  is  almost  completely  insensitive  to  changes  in  lens  position.  This  effect 
is  due  primarily  to  the  lack  of  activity  in  the  input  sectors  of  the  image. 


By  choosing  an  alternate  set  of  input  sectors  for  Image  5 and  evaluating  the  result- 
ant fluctuations  in  ac  power,  a curve  whose  sensitivity  to  focus  is  comparable  to 
those  of  the  other  images  is  obtained  (Figure  2-21).  The  alternate  sectors  used 
are  those  immediately  above  the  ones  shown  in  Figure  2-15.  The  average  activity 
increase  obtained  by  moving  the  input  areas  in  this  instance  is  33  percent.  The 
central  input  area  experiences  a two-fold  increase  in  activity. 


The  curves  relating  to  the  state  of  focus  of  the  targets  (Figures  2-22  through 
2-26)  exhibit  an  increased  sensitivity  to  focus  when  compared  to  the  background 
curves.  This  improvement  in  sensitivity  is,  again,  primarily  due  to  the  increased 
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Figure  2-16.  TOTAL  AC  POWER  - IMAGE  NO.  1 
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Figure  2-17.  TOTAL  AC  POWER  - IMAGE  NO.  2 
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Figure  2-20.  TOTAL  AC  POWER  - IMAGE  NO.  5 
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Figure  2-24.  TOTAL  AC  POWER  - IMAGE  NO.  3 TARGET 


Figure  2-25.  TOTAL  AC  POWER  - IMAGE  NO.  4 TARGET 
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Figure  2-26.  TOTAL  AC  POWER  - IMAGE  NO.  5 TARGET 

activity  of  the  target  sectors.  The  ac  power  of  the  target  sectors  is  about  twice 
that  of  the  average  background  sector.  The  target  curves  showing  the  most  sensi- 
tivity to  focus  are  those  from  Images  1,  2,  and  3.  The  targets  in  each  of  these 
images  are  small,  detailed,  high-contrast  objects.  Objects  of  this  type  appear  to 
provide  the  best  information  on  which  to  base  a state  of  focus  evaluation. 


To  determine  the  effect  of  focus  on  the  division  of  ac  energy  into  texture  and  edge 
activity,  the  image  descriptors  were  evaluated  on  representative  areas  of  selected 
FLIR  images.  The  test  FLIR  imagery  shown  in  Figure  2-2  falls  into  two  major  groups. 
One  group  (Images  1,  2,  and  3)  exhibits  small  targets  on  a relatively  bright  back- 
ground, while  the  other  group  (Images  4 and  5)  consists  of  larger  targets  against  a 
relatively  dark  background.  One  image  from  each  group  (Images  1 and  4)  was  selected 
to  represent  its  group  for  the  evaluation  of  the  image  descriptors.  Both  target  and 
background  areas  of  the  two  images  were  examined.  The  background  areas  are  identical 
to  those  shown  in  Figure  2-15.  For  target  input  information,  a 3 by  4 block  of  image 
sectors  was  centered  about  one  of  the  targets  in  each  image. 

The  curves  relating  the  behavior  of  texture  and  edge  activity  as  a function  of  lens 
position  are  shown  in  Figures  2-27  through  2-30.  It  can  be  seen  that  while  texture 
and  edge  activity  show  approximately  the  same  sensitivity  to  focus,  regardless  of  the 
type  of  input  data,  the  texture  curve  is  much  noisier.  Averaging  the  values  of  tex- 
ture activity  over  several  lens  cycles  would  probably  reduce  this  noise  level  substan- 
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Figure  2-29.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION  - 

IMAGE  NO.  4 BACKGROUND 


Figure  2-30.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION  - 

IMAGE  NO.  4 TARGET 
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tially,  but  the  increase  in  computation  time  for  such  an  approach  would  be  prohibitive 
in  a real-time  system.  For  the  background  type  data  seen  in  the  sample  FLIR  images 
provided  by  NVL,  the  division  of  activity  into  texture  and  edge  information  provides 
little  to  no  additional  information  about  the  image's  state  of  focus. 


The  ac  power  of  selected  image  sectors  can  provide  a useful  index  of  the  state  of 
focus  of  a FLIR  image.  The  sensitivity  of  the  measure  is  less  for  the  FLIR  images 
than  that  seen  on  the  test  pattern,  but  is  still  sufficient  to  form  the  basis  of 
an  error  signal  in  an  operational  servo  system. 


The  sensitivity  of  such  a measure  to  focus  is  seen  to  be  activity  dependent;  i.e., 
the  ac  power  of  a sector  does  not  perform  well  as  a focus  measure  when  the  maximum 
activity  of  the  sector,  regardless  of  focus,  is  low.  This  result  of  not  focusing 
well  on  low  activity  image  areas  agrees  with  the  manner  in  which  an  operator  would 
focus  an  image.  Rather  than  examining,  say,  the  sky  or  a dim  background  while 
adjusting  the  lens,  he  would  look  at  the  same  sharp  edge  or  fine  texture  to  deter- 
mine focus.  In  both  the  human  and  automatic  cases,  sufficient  ac  high-frequency 
energy  must  be  locally  present  to  evaluate  an  image's  state  of  focus. 

If,  in  the  case  of  automated  focusing,  each  subarea  of  the  image  in  question  cannot 
be  examined  in  a reasonable  time,  then  a criterion  of  minimal  activity  would  have  to 
be  established.  Only  those  subareas  satisfying  such  a criterion  would  then  be  con- 
sidered as  useful  input  data  to  the  focus  evaluating  device. 

The  division  of  activity  information  into  texture  and  edge  appears  to  be  of  little 
use  in  evaluating  the  state  of  focus  of  an  image.  The  behavior  of  both  texture 
and  edge  activity  as  well  as  those  of  the  other  image  descriptors  will  continue  to 
be  evaluated  in  subsequent  experiments. 

2.5  CONCLUSIONS 

| 

The  local  ac  power,  or  activity,  of  an  image  demonstrates  each  of  the  five  charac- 
teristics desired  of  a focus  treasure  presented  in  paragraph  2.1.  It  can  be  com- 
puted in  a time  span  that  is  compatible  with  a real  time  system;  its  sensitivity 


I 


2-38 


to  focus  can  rival  that  of  an  observer:  it  has  a good  signal-to-noise  ratio;  it 

exhibits  scene  independence,  subject  to  the  limitations  set  forth  a priori;  and, 
since  it  is  a simple  scalor  value  that  may  be  computed  for  each  image  frame,  it  is 
easily  adaptable  to  an  operational  system,  A real  time  imaging  system  that  has  an 
auto  focus  capability  is  certainly  feasible  and  easily  implementable  using  the  two 
dimensional  sensor  processor. 
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SECTION  3 
AUTO  DEBLUR 


3.1  INTRODUCTION 


Automatic  image  deblurring  seeks  a means  for  characterizing  the  blur  process, 
inverting  it,  and  deriving  a measure  of  the  residual  error.  Blur  characterization 
uses  physical  principles  to  narrow  the  anticipated  range  of  possibilities,  and  then 
applies  blind  deconvolution  to  arrive  at  a specific  form.  Blur  inversion  undoes  the 
degradation  up  to  the  point  that  ambient  noise  will  allow.  Error  measurement  uses 
either  a mean  square  error  (MSE)  criterion  or  the  set  of  image  descriptors,  each  with 
its  own  particular  merits. 


Blur  characterization  therefore  represents  a preliminary  step,  followed  by  error 
measurement.  Results  from  these  steps  set  the  ground  rules  and,  to  a certain  extent, 
provide  insight  in  addressing  the  inversion  problem.  Primary  attention  here  will  be 
devoted  to  the  first  two  steps. 

3.2  SPECIFIC  BLUR  TYPES 


Blurs  that  afflict  FLIR  systems  divide  into  several  types:* 

.2 


a.  Aperture  -J^  (kr)/kr  - e 


(r/r  )‘ 


b.  Scan  - rect 


(^)recc  (4) 


c .  Atmospheric  — *n  / — 


-■£)- 


(r/r  )‘ 
o 


Each  type  characterizes  the  limited  spatial  frequency  passband  of  an  element  in  the 
chain  of  transfer  functions  that  relate  the  original  object  (input)  to  the  final 


♦Additional  types  Include  depth-of-focus  and  image-motion  blurs.  These  will  be 
treated  as  time  permits.  Depth  of  focus  will  probably  receive  priority;  many 
others  are  currently  investigating  image  motion  effects. 
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imago  (output).  Together,  they  play  a role  somewhat  analogous  to  the  well-known 
transfer  function. 

Aperture  type  blurring  arises  from  the  failure  of  limited-sized  optics  to  capture  a 
sufficient  portion  of  the  object's  spatial  frequency  spectrum.  The  correspondingly 
limited  bandwidth  B does  not  permit  resolution  element  sizes  smaller  than  1/2B, 
using  classical  imaging  techniques.  This  result  derives  from  Nyquist's  Sampling 
Theorem  and  agrees  substantially  with  the  well-known  Rayleigh  criterion. 

Physically,  B varies  directly  with  aperture  size  D,  but  inversely  with  object- 
aperture  separation  z.  Consider  a one-dimensional  analog.  At  any  given  wavelength 
A>  the  object's  radiating  field  strength  g(x)  will  transform  to 

d;  - .(,*)  (1) 

" Fourier  transform  of  g(*>.) 

upon  reaching  the  aperture  plane.* 

An  aperture  that  admits  light  only  between  x = +D/2  alters  this  to  read 
g(~)  rect  £ 

Uz/  D (2) 

x ( 1,  for  |x|<D/2 

rect  — \ 

v 10,  otherwise 

Subsequent  optics  generally  seek  to  reverse  the  process  that  changed  g into  G. 

This,  however,  amounts  to  inverse  Fourier  transformation  of  the  above  expression, 
i.e. , 


Proof  of  this  relation  follows  from  expanding  the  quadratic  exponent  and  observing 
that  all  terms  other  than  the  cross  product  can  be  disregarded,  to  a first  approxi- 
mation. Relatively  small  objects  yield  non-zero  g(x)  only  over  a small  interval 

w ,At  Jar«f  values  of  z,  this  effectively  suppresses  the  quadratic  expo- 
ential  term  in  x . Likewise,  small  apertures  permit  elimination  of  x2.  Integra- 
tion of  g(x)  with  the  exponentiated  crossproduct  yields  the  above  transform  relation. 
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g(x) 


* sin  2^Bx 
21TBx 


Field  Strength  of 
Reconstructed  Image 


I «(*) I* 


2/sin  2ttBx\  2 intensity  of 


2ttBx 


/Reconstructed  Image 


B 


D 

21  z 


(3) 


One-dimensional  aperture-type  degradation,  therefore,  corresponds  to  a blur  function 
of  the  form  [sin  (...)/(...)]  . in  two  dimensions,  the  analogous  aperture  would  be 
a circular  hole  of  diameter  D.  A similar  derivation  would  yield  blur  function 

fJ1(2irBr)]2 

I 2rBr  | (4) 

(2  2 

Y - Wx  + y 

J.  » first  order  Bessel  function 


Scan-type  blur  characterizes  the  limited  resolution  resulting  from  sampling  the  output 
of  the  FLIR  scanning  mechanism.  This  image  acquisition  format  will  most  likely  be  en- 
countered in  the  second  generation  FLIR.  Horizontal  resolution  distance  can  be  expressed  as 


v = Horizontal  scan  velocity 


(5) 


Photoelectronic  bandwidth  - [2 (Sample  time)] 


while  vertical  resolution  distance  Ay  is  fixed  by  the  distance  between  scan  lines. 
These  considerations  dictate  the  blur  function 


v 
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A symmetrizing  approximation 


rect  rect (a^)  (7) 

however  may  be  preferred. 

Atmospheric  blur  arises  from  scattering  phenomena,  such  as  haze,  and  from  clear  air 
turbulence.  Scattering  generally  attenuates  all  radiation  wavelengths  smaller  than 
a given  cutoff  limit.  It  therefore  dictates  an  upper  limit  on  B in  equations  (3) 
and  (4)  . 

Clear  air  turbulence  produces  blurring  by  creating  random  spatial  and  temporal  den- 
sity variations.  The  resulting  spurious  diffraction  tends  to  warp  incident  wave- 
fronts  and,  generally,  to  degrade  mutual  coherence.  Mutual  coherence  expresses  the 
degree  to  which  neighboring  points  on  a reference  plane  across  incident  radiation 
designate  fixed  (if  unknown)  phase  relations  with  respect  to  one  another.  Drawing  a 
so-called  coherence  circle  about  a chosen  point  P in  this  plane,  the  phases  associ- 
ated with  all  interior  points  are  more  or  less  correlated  with  P,  while  exterior 
point  phases  exhibit  little  or  no  P-correlation. 

Coherence  circle  diameter  decreases  with  range.*  This  diameter  sets  a practical 

upper  limit  on  the  desired  aperture  size.  Larger  apertures  increase  image  intensity, 

but  not  resolution.  As  in  scattering,  therefore,  turbulence  introduces  no  new  types 

2 

of  blur  function.  Rather,  it  restricts  parameters  in  the  [J^  (•••)/(...)]  form 
associated  with  aperture-limited  blur. 

3.3  BLUR  SIMULATION 

Aperture-type  blurs  can  be  simulated  by  pinhole  masks  or  by  computer.  A pinhole 

mask  of  radius  r will  impose  a blur  of  the  form  in  (4)  with  B - r /Xf  when  intro- 
o o 

duced,  as  in  Figure  3-1. 


♦According  to  V.  I.  Tatalski’c  "Wave  Propagation  in  a Turbulent  Medium,"  the  diam- 
eter shrinks  as  the  -3/5  power  of  the  range.  Quantitative  aspects  of  his  theory  have 
been  disputed  by  later  authors,  but  the  general  shape  of  the  decreasing  trend  seems 
to  be  accepted. 
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Figure  3-1.  ANALOG  SCHEME  FOR  INTRODUCING  APERTURE-TYPE  BLURS 


Computer  implementation  would  approximate  (4)  by 


and  then  sample  it  uniformly  over  x and  y according  to  the  rule 

2 

2 2 2 2 2 2 r 

x + y - m (Ax)  + n (Ay)  £ max  (9) 

m,n  - integers 
Ax-  Ay  - linear  pel  size 


r 

max 


4/nB 


The  procedure  simplifies  the  form  and  spread  of  the  blur  function,  and  reduces  the 
result  to  a discrete  set  of  pel  values.  This,  in  turn,  permits  discrete  convolution 
of  original  image  with  blur  function  in  a format  suited  to  the  computer. 


3 . 4 EXPERIMENTAL  PROCEDURE 


These  considerations  form  the  basis  of  a quantitative  experimental  approach  to  blur 
simulation,  measurement,  and  removal.  Nominal  goals  Include: 
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• Quantitative  characterisation  of  naturally  occurring  blurs 

• Determination  of  practical  limits  on  deblur  improvements 

• Understanding  the  impact  of  real-time  constraints. 

Towards  these  ends,  the  experimental  procedure  will  follow  along  the  lines  of  Figure 
3-2. 

Transparencies  will  be  chosen  to  emphasize  four  major  types  of  features:  (1)  fine 

linear  details  with  horizontal  and  vertical  orientations,  (2)  fine  linear  details 
with  diagonal  orientation,  (3)  details  giving  rise  to  concentric-circular  patterns, 
and  (4)  finely  resolved  scenes  with  no  preferred  orientation.  This  will  include  at 
least  one  chart  whose  individual  test  patterns  emphasize  (1)  through  (4).  Current 

plans  call  for  computer  implementation  of  blur  simulation  on  a few  initial  sample 

scenes,  during  construction  of  the  analog  lens  scheme.  The  analog  scheme  can  then 

be  used  for  rapid  appraisal  of  a larger  number  of  scenes. 
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Figure  3-2.  GENERAL  EXPERIMENTAL  PLAN  FOR  AUTO  DEBLUR 


3-6 


Each  transparency  corresponds  to  about  120  pels  per  inch,  i.e.,  a resolution  distance 
of  0.02  cm.  Combined  with  a focal  distance  of  10  cm  and  a nominal  wavelength  of  0.3 
urn,  this  yields  a maximum  pinhole  diameter  of  150  ym.  Controlled  blurring  is 
achieved  by  cutting  the  actual  pinhole  size  to  fractions  of  this  maximum  value  (see 
Table  3-1).  Pinholes  of  these  sizes  are  readily  available  and  relatively  inexpensive. 


Table  3-1.  PINHOLE  SELECTION 


Pinhole  Diameter 
(um) 

Fraction  of 
D 

max 

Resolution  Length 
in  cm 

Resolution  Length 
in  Pels 

10 

1/15 

0.3176 

15 

25 

1/6 

0.1270 

6 

50 

1/3 

0.0635 

3 

100 

2/3 

0.0375 

1.5 

150 

1 

0.0200 

1 

Blur  function  determination  represents  a key  phase  of  the  experimental  program. 

Three  alternative  approaches  will  be  investigated.  These  include  an  empirical  search 
for  an  all-purpose  blur  function,  techniques  that  use  the  asymptotic  behavior  of  the 
Haar  coefficients,  and  techniques  that  attempt  simplifying  mathematical  approxi- 
mations . 

The  empirical  alternative  would  examine  the  spatial  frequency  bandwidths  of  a large 
number  of  scenes  that  have  all  been  subjected  to  the  same  specific  blur  simulation.’*' 
The  maximum  of  these  bandwidths  will  define  a pill-box  type  blur  function  of  diam- 
eter 1/2B  **.  This  result  can  be  compared  to  the  true  value  of  B,  which  character- 

izes  the  simulation.  The  comparison  may  indicate  near-equality,  a fixed  ratio,  or 
some  other  significant  relationship  between  the  two  quantities. 


*If  the  intended  use  implies  a characteristic  type  of  terrain,  visibility  or  other 
feature,  the  above  scenes  should  all  be  chosen  to  include  that  feature. 

**Some  authors  have  designated  the  pillbox  function  as  n (...);  in  this  case, 

*(2B  r)  where  r - Jxl  + y2. 
max  v J 
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The  asymptotic  approach  would  use  the  ability  of  the  Haar  expansion  to  generate 
local  functions  (higher  order  terms)  as  well  as  global  ones  (lower  order).  Because 
the  local  functions  describe  any  restricted  area  of  the  scene,  they  open  the  possi- 
bility of  superimposing  the  different  areas  in  such  a manner  that  scene  information 
will  decorrelate  and  wash  out,  while  blur  information  will  correlate  and  reinforce. 
This,  in  effect,  could  provide  a Haar  transform  approach  to  the  blind  deconvolution 
techniques  using  the  Fourier  transform  that  were  developed  by  Cole  and  Stockham  at 
the  University  of  Utah. 

Using  a one-dimensional  analog  for  simplicity,  coefficients  a(r,m)  of  Haar  expansion 


g(x)  mil  a(r,m)h(r,m,x)  (10) 

r m 

— j- 

where:  h(r,m,x)  » Haar  function  of  sequency  r and  offset  m*2 

can  be  grouped  into  subsets  a(r,l),  a(r,2),  a(r,3),  ...,  a(r,k),  ...  In  each  subset, 
the  magnitudes  of  the  a's  will  drop  off  beyond  some  high  sequency  cutoff  r - r^. 

The  average  value  and/or  linear  combinations  of  the  r^  can  then  be  compared  with  the 
resolution  distance  imposed  by  the  simulated  blur. 

The  third  approach  convolves  the  Haar  expansions  of  unblurred  figure  g^(x)  and  blur 
function  b(x)  to  obtain  an  expression  for  blurred  image  g2(x): 

g2(x)  ■ g1(x)*b(x)  = (pfk)  a(r>m)b(P>k) [h(r,m,x)*h(p,k,x)]  (11) 

Function  h (x)  can  be  regarded  as  a matched  filter  and  h (x)  as  the  candidate  wave- 
x P 

form.  Unless  k and  p refer  to  the  same  sequency,  the  convolution  should  produce 
zero  to  a first  approximation.  This  in  turn  should  yield 

k _r 

82(x)  * £ E a(r,m)b(r,k)  w[x  - (m,k)2  , r]  (12) 

r m“o 

where  w is  the  symmetric  W-shaped  waveform  resulting  from  the  convolution  of  an 
r-sequency  Haar  function  with  itself.  This  form  would  be  used  to  characterize  sub- 
sections of  the  image,  each  with  distinct  sets  of  a(r,m)  but  with  a common  set  of 
b(r,k).  Some  variation  on  the  usual  approach  to  blind  deconvolution  would  then  be 
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used  to  extract  the  b's.  These  results  can  be  compared  to  the  Haar  coefficients 
of  simulated  blur  actually  imposed. 

Deblurring  algorithms  will  generally  seek  to  retrieve  higher  order  portions  of  the 
Haar  spectrum,  which  have  been  suppressed  by  blurring.  Initial  experiments  will 
evaluate  the  effect  of  high  sequency  amplification  as  a restorative  process.  For 
these,  a noiseless  or  very  low  noise  environment  will  be  assumed.  Further  experi- 
ments will  then  account  for  realistic  noise  levels  through  a Haar  transform  approach 
to  optimum  filter  theory. 

This  theory  generally  proposes  direct  blur  inversion  in  regions  where  the  signal- 
to-noise  ratio  (SNR)  is  high,  but  further  deliberate  blurring  where  the  SNR  is  low. 
Different  settings  of  the  boundary  between  high  and  low  will  determine  whether  the 
result  will  show  good  deblurring  at  the  cost  of  high  noise  or  a more  moderate 
tradeoff. 

The  experimental  procedure  will  determine  the  Haar  image  spectrum  and  the  general 
spectral  level  of  the  noise.  The  regions  where  the  image  rises  above  the  noise  can 
then  be  identified.  A Haar  implementation  of  inverse  blurring  will  be  applied  to 
these  regions,  while  further  blurring  will  be  applied  to  the  alternate  (low  SNR) 
regions. 

3.5  MEASURFMENT  OF  RESULTS 

Quantitative  assessment  of  the  blurring  effect  can  use  either  mean  square  error 
(MSE)  or  image  descriptor  (ID)  criteria.  In  MSE,  a pel-by-pel  subtraction  between 
original  and  blurred  images  is  followed  by  squaring  and  summing.  This  numerical 
result  will  generally  increase  with  progressive  blurring.  An  intuitive  proof  would 
compare  the  original  image,  with  its  relatively  dense  forest  of  sampled  values, 
against  a thinned  version  corresponding  to  the  blurred  result.  With  greater  blurring, 
even  more  sample  values  would  be  deleted.  If  N represents  the  number  of  deleted 
values  at  any  stage  of  blurring,  then  a simple  statistical  argument  shows  that 


A B Bn 
o 


N 2 

(MSE)  - j gn 


(13) 


where: 


gn  = intensity  of  nth  pel  in  original  image 
A = Area  of  projected  image 


N 

o 


Original  bandwidth  and  number  of  samples 


AB  « Bandwidth  reduction  causing  blur 


In  terms  of  ID's,  the  expression  for  MSE  can  be  interpreted  as  the  activity  per  unit 
area.  Thus, 


(Activit 


N g 
on 


vitii 


Relative 

Resolution 

Degradation 


(14) 


3 . 6 FUTURE  EFFORTS 


Follow-on  goals  include  real-time  algorithms  that  derive  the  particular  blur  via 
blind  deconvolution,  enhancing  the  efficiency  of  these  algorithms  by  restricting 
their  search  to  (•••)/(•••)  and  rectangular  blur  types,  and  using  these  results 
to  derive  a real  time  deblurring  process.  Haar  processing  will  be  used  to  the 
extent  that  it  simplifies  these  tasks. 

Blind  deconvolution  essentially  multiplies  spectra  of  different  portions  of  the 
image  and  then  takes  the  kth  root  where  k = number  of  portions.  The  division  of  the 
image  into  16  by  16  pel  blocks  in  the  current  2D  Haar  processor  will  facilitate  this 
goal.  A further  advantage  will  arise  from  the  deterministic  nature  of  the  rectangu- 
lar blur.  In  the  final  deblurring  process,  tradeoffs  between  simplicity  and  the 
sacrifice  of  real  time  for  near-real  time  operation  will  be  studied. 
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